THE EULER CHARACTERISTIC OF LOCAL SYSTEMS ON 
THE MODULI OF CURVES AND ABELIAN VARIETIES 

OF GENUS THREE 

JONAS BERGSTROM AND GERARD VAN DER GEER 

Abstract. We show how to calculate the Euler characteristic of a local 
system Va associated to an irreducible representation V\ of the symplec- 
tic group of genus 3 on the moduli space M3 of curves of genus 3 and the 
moduli space ^3 of principally polarized abelian varieties of dimension 3. 



1. Introduction 

An irreducible representation V\ with highest weight A of the symplec- 
tic group GSp(2g, Q) defines a local system Va on the moduli space A g 
of principally polarized abelian varieties. Pull back under the Torelli mor- 
phism defines a local system V' A on the moduli space M. g of curves of genus 
g. The cohomology of such local systems is intimately connected with the 
cohomology of the moduli spaces M.g in of n-pointed curves and also with 
vector- valued Siegel modular forms of degree <?, cf., [3 [12], [9] . Therefore it 
is of some interest to be able to calculate the Euler characteristic of such a 
local system. 

Getzler showed in how to do this for M2 and Bini and the second 
author did this for local systems on the hyperelliptic locus H.3 of genus 3 in 
[5] by calculating the Euler numbers of a stratification. In the case of 
the non-hyperelliptic locus of M3, it seems difficult to calculate the Euler 
characteristics of the individual strata directly. Instead, we use the results 
of the first author (see [2]) on the motivic Euler characteristics of Y\ on 
for A of small weight, obtained by counting points over finite fields, to 
calculate linear relations between the Euler numbers of the strata. We were 
not able to determine the Euler numbers of all strata, but the information 
obtained suffices for determining the Euler characteristic of all local systems 
on M.3. To go from to A3 we need to calculate the Euler characteristics 
of the pull backs of our local systems to M.2 x A\ and to the locus Ai t 1,1 of 
products of three elliptic curves. We illustrate our results by giving tables 
of Euler characteristics on A13 and ^.3. We hope that this is a step towards 
a better understanding of the cohomology of local systems of genus 3. 
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We thank Torsten Ekedahl for providing us with the argument for the 
invariance of the Euler characteristic in Section [3l We also thank the Mittag- 
Leffler Institute for the hospitality enjoyed during the preparation of this 
paper. 

2. The Euler characteristic of a local system 

Let M3 be the moduli space of smooth genus 3 curves. It is a Deligne- 
Mumford stack of relative dimension 6 over Z. The universal curve ir 1 : 
A^3,i — > M3 defines, for any prime £, a natural £-adic local system R 1 ^^ 
of rank 6 on M3 <8> Z[l/£]. We shall denote this local system by ^V, or 
simply by V. It comes equipped with a non-degenerate symplectic pairing 
V x V Q/(-l). 

Similarly, we have the moduli space A3 of principally polarized abelian 
varieties of dimension 3, again a Deligne-Mumford stack of relative dimen- 
sion 6 over Z. The universal abelian threefold tt: X3 — > A3 also defines a 
natural ^-adic local system ^ir^Qi on A3 <g> Z[l/£], which we denote by /V, 
or simply by V. There is the Torelli morphism t3 : M3 — > A3 of degree 2 
between the stacks. The local system V on M3 <S>Z[l/£] is a pull back from 
the local system V on .A3 ® Z[l/£]. 

A partition A = {Ai > A2 > A3 > 0} of weight |A| = Ai + A2 + A3 
determines an irreducible representation V\ of Sp(6, Q) associated to A. We 
lift it to a representation of GSp(6, Q) with dominant weight (Ai — \2)la + 
(Ai — A2)76 + A37 c — \X\rj with j a , 7^ and 7 C suitable fundamental roots and 
rj the multiplier representation. Any such representation yields a symplectic 
local system Va of weight |A| on A3 <8> Z[l/£\, which appears 'for the first 
time' in the decomposition of 

Sym Al ~ A2 V Sym A2 ~ A3 (a 2 V) Sym A3 (a 3 V) , 

into irreducibles. If, for example, A = {Ai > > 0}, then Va = Sym Al (V). 
Similarly, we have a local system on M3 ®Z[l/£]. 

Our goal is to determine in an algorithmic way, for k = C or k = ¥ p , the 
Euler characteristic of compactly supported cohomology of V A on M3 and 
of Va on A3 . 

Compactly supported cohomology of a local system on a Deligne-Mumford 
stack presents various subtleties if the order of the automorphism groups is 
not invertible on the base, cf., the discussion in [TT] , Remarque 18.3.3. When 
the order of the automorphism groups is invertible one can manage with an 
ad hoc approach and that is what we do here. 

Suppose X is a Deligne-Mumford stack, tt: X — * X the natural map to 
its coarse moduli space, / : X — > S a map to a scheme S and g: X — > S 
the factorisation / = gn through the coarse moduli space. For T a (Q^-sheaf 
(with £ invertible on our base) on X we have Rir^J- = n^J 7 , since tt is finite, 
and hence by the Leray spectral sequence we have Rf*T = Rg^TV^F). We 
define the direct image with compact support by Rj\T := RgiXir^J 7 )- 
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With this definition we have Poincare duality for a local Q^-system T 
on X, if X is assumed to be smooth over some spatial base S and purely 
(i-dimensional. That is, we have a natural isomorphism 

RHom s (RfiF,Qe) = RU{F y {-d)[-2d\). 

Indeed, by the duality theorem for g we have 

KHom s (RfiT,Qe) = Rg*{RHom x (ir*F, g ] Q e )) . 

Now, g: X — > S is locally (in the etale topology) a quotient U/G — > S of a 
smooth morphism /i : U — ► i? of schemes by the action of a finite group G. 
Since /i is smooth we have h'Qe = d)[— 2d] and pushing down this sheaf 
to U/G and taking G-invariants gives g'Qe = Qe(—d)[—2d]. Furthermore, 
we may choose U to be an etale neighbourhood of an arbitrary point of 
X and since RHomu /g{ 7T *^ : \u /GiQi) — RH om u{ 7r *^ : \u,Qe) G we see that 
R7iomx('^*J 7 ,Qe) — ■ But since "K*T\u is a local system we find 

by taking G-invariants that (J ?r ) v = it*(J :y ). Putting these statements 
together gives us the desired duality formula. (Note that the sheaf tt^J 7 
need not be a local system on X.) 

We apply this to the stack by taking the compactly supported coho- 
mology of the direct image v*V'\ on the coarse moduli space M3 under the 
natural map v. M3 — > M3 and similarly for A3. We thus write 

12 

e c (M 3 ® k,V x ) = ^(-1)* dim Q£ i£(A* 3 ® fc, V A ) 
i=0 

and 

12 

e c (^ 3 ® fc,V A ) = ^2(-l) l dim Qe H l c (A 3 ® fc,V A ), 
i=0 

where H* means compactly supported £-adic etale cohomology, and Y\ 
means the local system iV\ with £ a prime different from the characteristic 
of our field. We shall see in the next paragraph that these Euler character- 
istics are independent of the choice of £, and in the next section that they 
are independent of k, which justifies our notation. 

In characteristic we also have the local system R 1 ^® on .M3 ® Q 
(resp. i? 1 ^^ on ^3 ® Q), denoted oV (resp. qV) and for each A, in the 
same way as in the £-adic case, an associated local system qY' x (resp. qY\). 
The comparison theorem 4.1 in [1] tells us that the Euler characteristic 
e c (A^3<8>C, qV'x) (resp. e c (^,3®C, qV a )) with respect to compactly supported 
topological cohomology is equal to the £-adic versions e c (A^3®C,V A ) (resp. 
6c (^3 <8> C, Y\)) defined above. 

3. Base change 

In this section we show that the Z-adic variants of our local systems 
commute with base change and we conclude that the Euler characteristic 
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e c (A43 (g) k,Y' x ) is independent of k being equal to C or ¥ p , for any p differ- 
ent from £. 

Theorem 3.1. Let f : X — > S be a smooth, proper, relatively Deligne- 
Mumford (cf., |17l 7.3.3] J map between algebraic stacks such that the prime 
I is invertible on S. Suppose given a relative normal crossing divisor D C X 
and a locally constant sheaf £ ofZe, Qe or t-torsion modules on U := X\D. 
If £ is tamely ramified along D, then the R l g*£ are locally constant and com- 
mute with base change where g is the structure map g: U — > S. 

Proof. We start by showing that if j : U — > X is the open embedding, then 
Rj*£ commutes with base change on S and is constructible. Indeed, this is 
a local statement on X so we may assume that X is a scheme and then it 
follows from [15^ Th. finitude, 1.3.3]. For / we have the proper base change 
theorem [1TL Thm. 18.5.1], using [17, Thm. 16.6]. Hence, we conclude that 
Rg*£ = Rf*Rj*£ commutes with base change and is constructible. 

Furthermore, the smooth base change theorem [TJ Cor. 1.2] for / is true 
as it is local on X. The fact that K L g )f £ is locally constant now follows as 
in the proof of [lj Thm 2.1]. Indeed, properness is used only through the 
proper base change theorem (in fact only to reduce to the case when the 
base is normal) and, in turn, it is only applied to show that R l g*£ commutes 
with base change. □ 

Corollary 3.2. Let p: Ai g — > Spec(Z[l/£|) be the structure map and ir : 
■Mg i — > Mg the universal curve. If V is a locally constant system on A4 g 
then R l p\V is a locally constant system over Spec(Z[l/^]) that commutes 
with base change. 

Proof. The statement follows from the corresponding statement for R l p*V 
by duality (see Section 2) and the case of R l p*V follows from Theorem 13.11 
applied to M. g — > Spec(Z[l/£]) once one has verified tameness. By purity 
and Abhyankar's lemma |13l X.3.6], it is enough to verify this at the local- 
ization of Ai g at a generic point of a component of the boundary. That 
generic point is however of characteristic 0. □ 

In view of this independence of the characteristic we shall often just write 
e c (A^3,V' A ) instead of e c (Ms <g> k,Y' x ) and c c (.43,Va) instead of e c (As <g> 
k,Y\). The invariance of the Euler characteristic for ^3 follows from the 
stratification of ^.3 (= £3(^3) U x -4i U found in the beginning 

of Section [HJ 

4. Calculating the Euler characteristics 

We now turn to the calculation of the Euler characteristics. The first 
remark is that the Euler characteristics are unchanged if we change the 
representation A by a power of the multiplier rj (twisting). We can therefore 
restrict our attention from GSp(6,Q) to Sp(6, Q). 
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In this section we will work over C and write .M3 for Mz (8> C The 
local system V' A will stand for oV' x . The Euler characteristic e c (Ais,Y' x ) 
is calculated by descending from the stack .M3 to the coarse moduli space 
M3 under the natural map fj,: M.3 — > M3 and by calculating e c (M%, /i*(V A )) 
using the stratification of M3 (8) C by the automorphism group of the genus 
3 curve. On each stratum the automorphism group of every curve 

is equal to G, and the direct image /i*(V) is a local system. The Euler 
characteristic is then given by (see, e.g., [BJ Theorem 5.13], and also [IB] ) 

(4.1) e c (M 3 ,Y' x ) = ]Te c (£(G))dim(V A G ), 

G 

where e c (£((?)) is the topological Euler characteristic of £(G) and V A G is 
the space of G-invariants. 

The dimension of the invariant part V A G is determined as follows. The 
action of G on the cohomology group H l (C, Q) of a curve C of genus 3 
defines a homomorphism r: G — > Sp(6, Q). We denote the eigenvalues of 

Kt) by 

{«7 

Let be the complete symmetric function of degree d in six variables. For 
the proof of the following fact we refer to |1CH Prop. 24.22]. 

Proposition 4.1. If J\ is the determinant of the 3x3 matrix whose i-th 
row is 

and 

J,i(a 7 , 6 7 ,c 7 ), 

i/ien we have ^ 

dim(V A G ) = — A(a 7 ,6 7 ,c 7 ). 
WLr yea 

Hence, we can calculate the Euler characteristics of the local systems 
V'\ on Ai% if we have the strata and the Euler numbers of the strata on 
M3. It seems very difficult to determine these Euler numbers directly, but 
by using our moduli spaces over finite fields we shall determine sufficiently 
many linear relations between them. 

5. The stratification of the moduli of non-hyperelliptic curves 

Also in this section we write M3 for ,M3<g>C and V A for oV'y The calcula- 
tion of the Euler characteristic of V A on M3 is done by using a stratification 
on M3 (and on M 3 ). First we have M3 = U Hz with Hz the hyperel- 
liptic locus and M\ the locus of non-hyperelliptic curves of genus 3. Since 
an algorithm for the calculation of the Euler characteristic of V A on Hz 
was given in [5] we are left with calculating e c (Ai^,Y' x ). For this we use 
the stratification of Ai® given by the automorphism group in the manner 
described above. 
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A non-hyperelliptic curve C of genus 3 over the field of complex numbers 
C can be given as a smooth projective curve of degree 4 in the projective 
plane, the canonical image of C. The automorphism group acts on the vector 
space H (C,Qq) and thus induces and is in fact induced by a projective 
automorphism of the projective space of H°(C, Sl^). It is well-known which 
groups occur as an automorphism group of a non-hyperelliptic curve of genus 
3. We list the possibilities in Table [TJ This list is found in [20J, cf. also |19j . 



Table 1. The automorphism groups 



i 


G 


#G 


Normal form: / 


dim(S(G)) 





1 


1 


/ 


6 


1 


Z/2Z 


2 


x 4 + x 2 f(x,y) + g(y,z) 


4 


2 


Vi 


4 


x 4 + y 4 + z 4 + ax 2 y 2 + by 2 z 2 + cx 2 z 2 


3 


3 


Z/3Z 


3 


yz 3 + x(x — y)(x — ay){x — by) 


2 


4 


s 3 


6 


x 3 y + y 3 z + x 2 y 2 + axyz 2 + bz 4 


2 


5 


D 4 


8 


x 4 + y 4 + z 4 + ax 2 y 2 + bxyz 2 


2 


6 


Z/6Z 


6 


z y + x + ax y + y 


1 


7 


Tie 


16 


x 4 + y 4 + z 4 + arr 2 ?/ 2 


1 


8 


S 4 


24 


+ y 4 + z 4 + a(x 2 y 2 + x 2 z 2 + y 2 z 2 ) 


1 


9 


Z/9Z 


9 


x 4 + rry 3 + yz 3 





10 




48 


x 4 + y 4 + z 4 + (4C 3 + 2)x 2 y 2 





11 


T96 


96 


4 i 4 i 4 

x + y + z 





12 


Tl68 


168 


x 3 y + y 3 z + z 3 x 






Accordingly, the moduli space M.^ carries a stratification indexed by the 
automorphism groups of non-hyperelliptic curves of genus 3. There are 
thirteen (open) strata T,c i and the inclusion relations between the closures 
of the strata are given in Table [2j 

Table 2. The diagram of strata 



G » 




The groups Gi are listed in [20J as subgroups of PGL3 acting on the co- 
ordinates x, y, z of the normal forms found in Tabled) In view of Prop. 14.11 
we need the representations p(Gi) on the space of differentials H°(C, Oj), or 
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more precisely, the eigenvalues of all the group elements. These representa- 
tions are listed in Table EJ where the standard basis is identified (on z ^ 0) 
with u, yi u;, y 2 u, where y x := x/z, y 2 := y/z, G(y 1 ,y 2 ) := /(2/1, 2/2,1) and 
u; is the differential found by gluing dy\/ {dG / dy 2 ) and dy 2 / {dG / dy\) . 

The notation for Table Q] and [3] is as follows: V4 is the Klein four-group; 
-D4 is the dihedral group of order 8; S n denotes the symmetric group on n 
letters; T n denotes a group of order n with r^s = SL(3,F2); Q n is a n-th 
root of unity. 

Table 3. The eigenvalues 



Generators of p(G) 



9 
10 

11 
12 



6 
16 

24 
9 
48 

96 
168 



1 



diag(-l,l,-l) 
diag(-l,l,-l), diag(-l,-l,l) 
diag(Cf,C3,C3) 



















-1 


n 


i 





N 











-1 


/ 



diag(-C|,C3, -C3) 

diag(-l,l,-l), diag(l, ( 

,0 1 

'0 1 N 
10 

1 0; 

dia| 

0' 

Cs Cf h 775 





1 

Cr 2ij )/( 







-i), 







(: 







1 


) S,9> 


CO 










n 










diag(C 7 ,C 7 4 ,C 7 '), (C 7 




6. Linear relations between the Euler numbers 

In this section we use information obtained by counting points over finite 
fields to get linear relations between the Euler numbers of the strata. 

We consider the moduli space Ai® over Z[l/2, l/£j. By the work of the 
first author the Euler characteristic a\ = e c (Ai^ (g> ¥ p ,Y' x ) are known for 
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low weight local systems. More specifically, in [2], for |A| < 7, the trace of 
Frobenius on e c (M.^ <S> k,Y' x ) is computed for all finite fields k, and these 
traces turn out to be given by a polynomial in the cardinality of the field. 
By Deligne's proof of the Weil conjectures (see [8j) this implies that the 
Euler characteristic is obtained by substituting 1 for the cardinality of the 
field in this polynomial. The results for the traces, in the cases |A| = 6,7, 
are presented in Theorems 16.2 and 16.3 of [2]. For the results in terms of 
Euler characteristics see Table H] and Table [5j 

Since the Euler characteristics e c {M^®k, V A ) for any field k of character- 
istic different from 2 are the same this leads, by (|4,ip . to a system of linear 
equations in the unknowns e, := e c (£G 4 ) 

12 

^dhn(V? 4 )ei = o A . 
i=0 

We shall use these equations for all A with |A| < 4 and A = (6,0,0). We 
obtain in this way the following result. 

Proposition 6.1. The Euler numbers ei = e(Ej-) of the strata Ej on A^3<8)C 
satisfy the following equations: 

ex = -3e , e 2 = 2e + e 8 + 1, e 4 = -e 8 - 1, e 5 = -e 8 , 

and 

£3 = 0, e 6 = e 7 = -1, e 9 = e w = en = e 12 = 1. 

Note that this proposition agrees with the fact that each of the strata 
Sj consists of one point for i = 9, 10, 11 and 12. These equations do not 
determine the Euler numbers e^, but there are some easy relations between 
the linear equations in the Euler numbers that reduce the number of 
unknowns. If we write 

ki(X) := dim(vf ) 

and substitute the relations of Proposition 16.11 in Yll=o ki(\)ei, then the 
coefficients of eo and e 8 are Co = ^o(^) — 3&i(A) + 2fc 2 (A) and c 8 = /^(A) — 
fe4(A)— /c5(A)+/c 8 (A). The following lemma shows that cq = and — co+2c 8 = 
0, i.e., the sum Xa=o ki{X)ei does not depend upon eo and e 8 . 

Lemma 6.2. Let ki(\) = dimlY^). Then we have the relations 

k (\)-3k 1 (\) + 2k 2 (\) = 0, 

and 

-ko(X) + 3 fci(A) - 2 k 4 (X) -2k 5 (X) + 2 k 8 (X) = 0. 
Proof. We use the formula 

dim(V^) = — J\(a^,b y ,c y ), 







where {a 7 , & 7 , c 7 , a~ l , b~ x , c^ 1 } are the eigenvalues of 7 on the standard rep- 
resentation. The lemma now follows from a glance at the table of eigen- 
values from which it follows that the weighted sum of eigenvalues agree. 
The first relation is, for example, clear from the fact that the eigenval- 
ues for the groups Go,Gi and G2 are {[1,1,1]}, {[1, 1, 1], [— 1, — 1, 1]} and 
{[1,1,1], [-1,-1,1], [-1,-1,1], [-1,-1,1]}. " □ 
Combining the preceding lemma and proposition gives the following re- 
sult. 

Proposition 6.3. For i = 0, ... ,12 we let fcj(A) := dim(V^ 1 ). Then the 
Euler characteristic ofY' x on the non-hyperelliptic locus Ai® is given by 

12 

e c (MlY' x ) = k 2 (X) - k A (X) - k 6 (X) - k 7 (\) +^> i (A). 

j=9 

By the additivity of the Euler characteristic we get the Euler character- 
istic on A^3 by adding the contribution from the hyperelliptic locus. An 
algorithm for determining the values e c (7i.3,Y' x ) was given in [5]. 

Remark 6.4. Note that for local systems of odd weight |A| the Euler 
characteristic vanishes on the hyperelliptic locus H.3 in view of the presence 
of the hyperelliptic involution. We therefore present results on the Euler 
characteristic of .M3 for local systems of even and odd weight separately in 
Table [5] and Table [5j Note that the values we obtain for the Euler char- 
acteristics on the hyperelliptic locus (and its complement) are only valid if 
the characteristic is either or greater than 2. The closure of H3 in M3 is 
namely singular in characteristic 2, along the closure of the strata consisting 
of a rational backbone with three elliptic tails, c.f. [4j Theorem 2.8]. For 
examples where the Euler characteristic behaves differently in characteristic 
2, see p] or [3]). 

6.1. The hyperelliptic case and the case g = 2. It is interesting to apply 
the method used here for the non-hyperelliptic case to the hyperelliptic case. 
We have a stratification 7i% by strata parametrized by the possibilities 
Hi, i = 1,...,11, for the automorphism group of a hyperelliptic curve, cf., 
[5]. The values b\ = e c {TL3®¥ p , Y' x ) for |A| < 6, obtained by counting points 
over finite fields (see [3]) give us a system of linear equations for the Euler 
numbers <?j = eiT^Hi) with a 3-dimensional solution space. Just as for the 
genus 3 non-hyperelliptic case, the Euler characteristics do not depend on 
these remaining three parameters. 

If the same procedure is applied to M2 you actually find the Euler char- 
acteristics of all individual strata from the values c\ = e c (7i.2 <8> F p , V^) for 
|A| < 6. 
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7. Branching 

In order to calculate the Euler characteristic on ^.3 we need branching 
formulas for representations of the symplectic group. Since the Euler char- 
acteristic of Y\ is unchanged by twisting with the multiplier representation, 
we may and do work with Sp(6) := Sp(6, Q) instead of GSp(6,Q). 

Let U be a finite-dimensional irreducible complex representation of the 
group G := Sp(2) 3 x S3, where S3 denotes the symmetric group of the three 
factors Sp(2) := SL(2, Q). We will need branching formulas from Sp(6) to G. 
In the literature we found formulas for branching from Sp(6) to Sp(2) 3 (see, 
e.g., [18]), but none to G. The restriction of U to Sp(2) 3 decomposes as a 
direct sum U = ©jg/Wj of irreducible representations of Sp(2) 3 . The group 
S3 acts transitively on the index set. Let E = {a £ S3 : a(W) = W} be the 
stabilizer of a component W = Wi. Then W is an irreducible representation 
of the semi-direct product Sp(2) 3 x E, hence of the form V a M Vg M Vy with a 
E-action, where V n for n > denotes the irreducible representation of rank 
n + 1 of Sp(2). If E contains a 3-cycle then a = (3 = 7 and if E contains a 
2-cycle then #{a, (3, 7} < 2. 

If E is trivial then #1 = 6 and U consists of six copies of V a M Vg M V*,. 
In this case U is obtained by inducing the representation V a Kl Vg M from 
Sp(2) 3 to G. We denote it by R Q ^^. It is independent of the ordering of 
a, (3, 7 and therefore we shall assume that a > f3 > 7. 

Next, suppose that E is of order 2. Then W is of the form V a El Vp Vp 
and E acts on it, making it a representation of Sp(2) 3 x E. There are two 
possibilities for the action of a generator of E: 

uMvMw^uMwMv or — uM w M v 

and U is then the induced representation from Sp(2) 3 x E to G. We denote 
the two possibilities by R^g an d R^g- Note that a and /3 need not be 
different. 

If E is of order 3 then a = (3 = 7 and a generator a of E acts on 
V a ^V a M V a via 

with p a primitive third root of 1 and e = 0, 1 or 2. The representation U is 
then the induced representation from Sp(2) 3 x E to G. If e = we denote 
it by T a , while for e = 1 and 2 the result is the same and is denoted by T' a . 
Finally, if E = S3 then U is V a Kl V a M V a with the action of §3 given by 

Ml M U2 E U3 U c - in) U a - 1(2) ^ U a - 1(3) or 

sgn(o-)ii CT -i(i) ^1^-1(2) El i^-ip). 

We denote the two representations by and R~ . 
We have the following two relations: 

T a = Rq © R a and Ra,a = -^a © Ra • 
Thus we obtain the following result. 
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Lemma 7.1. Every irreducible representation of Sp(2) 3 x S3 is a virtual 
sum of representations of the form R a ^^, R^ a and R^. 

The exterior products A*Vi 0,0 f° r i = 1, 2, 3 of the standard representation 
of Sp(6) form a basis for the representation ring of Sp(6). Their restrictions 
to the subgroup Sp(2) 3 x S3 are given by 

Viflfi\G = Rifi, A 2 V\fi,o\G = R 01 (B Rq , A 3 Vi 5 o,o|G = Rf^ © -Ri,o,o- 

Explicitly, the representation V\ corresponds to the symmetric function J\ 
and by writing J\ in terms of the elementary symmetric functions e±, e 2 and 
e3 one finds the expression of V\ in terms of the three exterior products. 
Thus we can determine the branching completely. We implemented this in 
a computer program. 

8. The Euler characteristics on A3 

In this section we will use the stratification, 

A3 = h(M° 3 ) u t 3 (H 3 ) U t 2 (M 2 ) x Ai U Ai,i,i, 

to compute e c {A$, V\) for any partition A, where t g : M. g — > A g is the Torelli 
morphism. For hyper elliptic curves the Torelli morphism has degree one 
and hence e c {tziji. 3 ) Va) (resp. e c (t2(M.2), Va)) is equal to e c (H3,Y' x ) (resp. 
e c{M-2, V A )). On the locus in M3 of non-hyperelliptic curves the Torelli 
morphism has degree two, due to the existence of the automorphism "—1" on 
the Jacobians. That is, if G is the automorphism group of a non-hyperelliptic 
curve C, then G' := {±g : g G G} is the automorphism group of its Jacobian. 
The canonical isomorphism between if 1 (C) and -fT 1 (J(C)) and the fact that 
— 1 £ G' acts as —1 on i7 1 (J(C)) then shows how the elements of G' act on 
H l (J(C)). The Torelli morphism is a homeomorphism of the coarse moduli 
spaces and thus the Euler characteristic of the strata of £3(^3) are the 
same as the corresponding strata of .M3. Since the polynomials J\ are even 
(resp. odd) if A is even (resp. odd), we find by applying formula (14. lj) . that 
Zcitsi-M^), Va) is equal to e c (Ai 3 , V' A ) if A is even and zero if A is odd. 

We will apply a Kunneth formula to compute e c {t2{M.2) x A\, Va). For 
this we need to know the branching, that is the restriction of the repre- 
sentation Va of Sp(6,Q) to Sp 4j2 := Sp(4,Q) x Sp(2,Q). If y A |s P4i2 = 
M) „(V^ H V u )® m ^ it follows that 

e c (t 2 (M 2 ) X Ai,V x ) =J2 m w e c{M 2 ,t*Y tl )e c (Ai,Y u ). 

A formula for the branching from Sp(6, Q) to Sp 4 2 (i-e., for the numbers 
m^u) can for instance be found in [18] . 

Finally, to compute e c {A\^\,Y\) we will also use branching, in this case 
from Sp(6, Q) to G = Sp(2,Q) 3 xi S3, where the symmetric group S3 acts 
by permuting the three factors Sp(2,Q). This branching was treated in 
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the preceding section. Since Ai t i,i = (A.i) 3 /§3, we have to compute the 
invariant part of the cohomology of the local system Y\ restricted to (A\) 3 . 
By Lemma 17. II we can write Vx|G, as a virtual representation, 

m aAl R a ,(3,~,+Y^ ( m a,/3 R t(S + m a,f3 K,p) +Y1 i™* R u + m a K) ■ 

a>/3>j a,/3 a 

We then have 

(A ) V a )e c (^i,V^)e c (^i,V 7 ) 

a>/3>7 




Thus our knowledge of the Euler characteristics of local systems for g = 1 
and g = 2 together with those of local systems on suffices to calculate 
algorithmically the Euler characteristics on .A3. 

We calculated these for all A with |A| < 60, see Table [3 for the results for 
|A| < 18. A first check is that the value e c (A3,V\) = 5 for the trivial local 
system given by A = (0, 0, 0) agrees with a result of Hain, who calculated the 
rational cohomology of A3, cf., |14j . A further indication of the correctness, 
besides the fact that while we are summing rational numbers we always 
found integer values, is that the absolute value of the Euler characteristic of 
V\ on .M3 is in general much smaller than the Euler characteristic of on 
the hyperelliptic locus and its complement. This is illustrated in Table HI 
A similar phenomenon was observed for the genus g = 2 case (cf., [9]) and 
can be observed for A3 too, as illustrated in Table [6j 
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Table 4. Euler characteristics of V' A (even weight) on the 
parts of the stratification of M 3 . 



A 


n 3 




A A 

M 3 


A 


n 3 




A A 

M3 


(0,0,0) 


1 


2 


3 


(5,2,1) 


-10 


6 


-4 


(2,0,0) 


-1 


1 





(4,4,0) 


-5 


7 


2 


(1,1,0) 





-1 


-1 


(4,3,1) 


-4 


4 





(4,0,0) 


-1 


1 





(4,2,2) 


-7 


8 


1 


(3, 1,0) 











(3,3,2) 


-2 





-2 


(2,2,0) 


-1 


1 





(10,0,0) 


-17 


13 


-4 


(2,1,1) 





1 


1 


(9, 1,0) 


-22 


20 


-2 


(6, 0, 0) 


—5 


4 


— 1 


(8, 2, 0) 


—43 


37 


—6 


(5,1,0) 


-2 


1 


-1 


(8,1,1) 


-8 


7 


-1 


(4,2,0) 


-5 


3 


-2 


(7,3,0) 


-34 


26 


-8 


(4,1,1) 





-1 


-1 


(7,2,1) 


-32 


28 


-4 


(3,3,0) 





2 


2 


(6,4,0) 


-37 


31 


-6 


(3,2,1) 











(6,3,1) 


-26 


17 


-9 


(2,2,2) 


-3 


4 


1 


(6,2,2) 


-27 


25 


-2 


(8,0,0) 


-7 


9 


2 


(5,5,0) 


-6 


8 


2 


(7,1,0) 


-8 


7 


-1 


(5,4,1) 


-22 


22 





(6,2,0) 


-13 


14 


1 


(5,3,2) 


-12 


9 


-3 


(6,1,1) 


-2 





-2 


(4,4,2) 


-15 


13 


-2 


(5,3,0) 


-10 


8 


-2 


(4,3,3) 





-2 


-2 



Table 5. Euler characteristics of V' A (odd weight) on M 3 . 



A 


M 3 


A 


M 3 


A 


M 3 


A 


M 3 


(1,0,0) 





(2,2,1) 





(3,2,2) 





(5,3,1) 


20 


(3,0,0) 





(7,0,0) 


-2 


(9,0,0) 


4 


(5,2,2) 


2 


(2,1,0) 





(6,1,0) 


4 


(8,1,0) 


8 


(4,4,1) 


2 


(1,1,1) 





(5,2,0) 


4 


(7,2,0) 


10 


(4,3,2) 


4 


(5,0,0) 





(5,1,1) 


10 


(7,1,1) 


18 


(3,3,3) 


8 


(4,1,0) 


2 


(4,3,0) 


2 


(6,3,0) 


20 


(11,0,0) 


4 


(3,2,0) 


-2 


(4,2,1) 


2 


(6,2,1) 


12 


(10,1,0) 


30 


(3,1,1) 


2 


(3,3,1) 


4 


(5,4,0) 


6 


(9,2,0) 


36 



Table 6. Euler characteristics of Va (of high weight) on the 
parts of the stratification of ^,3. 



A 


n 3 


Mi 


M 2 x Ai 


Am 


A3 


(40,0,0) 


-3825 


3257 


731 


-161 


2 


(32,5,3) 


-188587 


156651 


44843 


-12615 


292 


(24,12,4) 


-502970 


419733 


116703 


-32415 


1051 


(21,15,4) 


-351374 


292508 


82372 


-22910 


596 


(14, 13,13) 


-2262 


1795 


649 


-191 


-9 
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Table 7. Euler characteristics of Va on A3. 



\ 

A 


A„ 


\ 

A 


A„ 
•^3 


\ 

A 


A„ 
«^3 


\ 

A 


A„ 


(0,0,0) 


5 


(4,3,3) 





(7,6,1) 





(18,0,0) 


-5 


(2,0,0) 


-2 


(12,0,0) 


-1 


(7,5,2) 


1 


(17, 1,0) 


-7 


(1; 1)0) 


-1 


(11,1,0) 


-3 


(7,4,3) 





(16, 2, 0) 


-4 


(4,0,0) 


—2 


(10, 2, 0) 





(6,6,2) 


4 


(16, 1, 1) 


—1 


(3, 1,0) 





(10,1,1) 


-1 


(6, 5, 3) 


1 


(15,3,0) 





(2, 2, 0) 





(9,3,0) 





(6,4,4) 





(15, 2, 1) 





(2, 1, 1) 


1 


(9, 2, 1) 





(5,5,4) 





(14, 4, 0) 


—4 


(6,0,0) 


-3 


(8,4,0) 





(16,0,0) 


—2 


(14,3, 1) 


—4 


(5, 1,0) 


—1 


(8, 3, 1) 


—2 


(15, 1,0) 


—4 


(14, 2, 2) 


7 


(4, 2, 0) 





(8, 2, 2) 


3 


(14, 2, 0) 


4 


(13, 5, 0) 





(4, 1, 1) 


1 


(7,5,0) 


—2 


(14, 1, 1) 


—1 


(13,4, 1) 





(3, 3, 0) 





(7, 4, 1) 





(13, 3, 0) 


—4 


(13, 3, 2) 


1 


(3, 2, 1) 





(7, 3, 2) 


1 


(13, 2, 1) 





(12, 6, 0) 


3 


(2, 2, 2) 


1 


(6,6,0) 


5 


(12,4,0) 


4 


(12, 5, 1) 


-3 


(8, 0, 0) 





(6, 5, 1) 


—1 


(12, 3, 1) 


—3 


(12, 4, 2) 


4 


(7, 1,0) 


—1 


(6,4,2) 





(12, 2, 2) 


4 


(12,3,3) 


— 1 


(6,2,0) 


1 


(6,3,3) 


1 


(11, 5, 0) 


—2 


(11, 7, 0) 


-3 


(6, 1, 1) 





(5, 5, 2) 


—1 


(11, 4, 1) 





(11,6, 1) 





(5,3,0) 





(5,4,3) 





(11,3,2) 


—1 


(11, 5, 2) 


7 


(5, 2, 1) 





(4, 4, 4) 





(10,6,0) 


4 


(11,4,3) 





(4, 4, 0) 





(14,0,0) 


—2 


(10, 5, 1) 


—2 


(10,8,0) 





(4, 3, 1) 





(13, 1, 0) 


-3 


(10,4,2) 


4 


(10, 7, 1) 


-1 


(4, 2, 2) 


1 


(12, 2, 0) 


-1 


(10,3,3) 





(10,6,2) 


5 


(3, 3, 2) 





(12,1,1) 





(9, 7, 0) 





(10, 5, 3) 





(10,0,0) 


-4 


(11,3,0) 


-2 


(9,6, 1) 





(10,4,4) 


4 


(9, 1,0) 





(11,2, 1) 





(9, 5, 2) 





(9,9,0) 


4 


(0, 2, OJ 





/in A n\ 
(10, 4, OJ 


4 


(9, 4, 6) 





( a c 1 \ 

(9,8, 1) 





(8,1,1) 


1 


(10,3,1) 


-2 


(8,8,0) 


8 


(9,7,2) 





(7,3,0) 


-2 


(10,2,2) 


1 


(8,7,1) 


1 


(9,6,3) 


8 


(7,2,1) 





(9,5,0) 





(8,6,2) 


3 


(9,5,4) 





(6,4,0) 





(9,4,1) 





(8,5,3) 





(8,8,2) 


3 


(6,3,1) 


-1 


(9,3,2) 





(8,4,4) 


8 


(8,7,3) 


-3 


(6,2,2) 





(8,6,0) 





(7, 7,2) 





(8,6,4) 





(5,5,0) 





(8,5,1) 


-2 


(7,6,3) 





(8,5,5) 





(5,4,1) 





(8,4,2) 





(7,5,4) 





(7,7,4) 


-1 


(5,3,2) 


-1 


(8,3,3) 





(6,6,4) 


3 


(7,6,5) 





(4,4,2) 





(7, 7,0) 


-1 


(6,5,5) 





(6,6,6) 
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